Xbasic

Overview of Regular Expressions

Description

An overview of regular expressions, including a look at literals, wildcards, repeats, and more.

Portions of this page are obsolete.

Literals

All characters are literals except: ., |, *, ?, +, (, ), {, }, [, ], ^, $ and \. These characters are literals when preceded by a "\". A literal is a character that matches itself, or matches the result of traits_type::translate(), where traits_type is the traits template parameter to class reg_expression.

Wildcard

The dot character "." matches any single character except when:

match_not_dot_NULL is passed to the matching algorithms, the dot does not match a NULL character;
match_not_dot_newline is passed to the matching algorithms, then the dot does not match a newline character.

Repeats

A repeat is an expression that is repeated an arbitrary number of times.

An expression followed by * can be repeated any number of times including zero.
An expression followed by + can be repeated any number of times, but at least once, if the expression is compiled with the flag regbase::bk_plus_qm then + is an ordinary character and \+ represents a repeat of once or more.
An expression followed by ? may be repeated zero or one times only, if the expression is compiled with the flag regbase::bk_plus_qm then "?" is an ordinary character and \? represents the repeat zero or once operator.
When it is necessary to specify the minimum and maximum number of repeats explicitly, the bounds operator {} may be used, thus a{2} is the letter a repeated exactly twice, a{2,4} represents the letter a repeated between 2 and 4 times, and a{2,} represents the letter a repeated at least twice with no upper limit. Note that there must be no white-space inside the {}, and there is no upper limit on the values of the lower and upper bounds. When the expression is compiled with the flag regbase::bk_braces then { and } are ordinary characters and \{ and \} are used to delimit bounds instead. All repeat expressions refer to the shortest possible previous sub-expression: a single character; a character set, or a sub-expression grouped with () for example.

Examples

Example: Result
ba*: Matches any string that begins with 'b' followed by zero or more 'a's. Eg, b, ba, baaa
ba+: Matches any string that beings with 'b' followed by one or more 'a's. Does not match 'b'. Eg, ba, baaaa
ba?: Matches 'b' or 'ba'.
ba{2,4}: Matches a 'b' followed by two, three, or four 'a': 'baa', 'baaa' and 'baaaa'.

Non-greedy repeats

Whenever the "extended" regular expression syntax (the default) is in use then non-greedy repeats are possible by appending a '?' after the repeat; a non-greedy repeat is one which will match the shortest possible string.

For example to match html tag pairs one could use something like:

<\s*tagname[^>]*>(.*?)<\s*/tagname\s*>

In this item $1 will contain the text between the tag pairs, and will be the shortest possible matching string.

Parenthesis

Parentheses serve two purposes, to group items together into a sub-expression, and to mark what generated the match. For example the expression (ab)* would match all of the string ababab.

Non-Marking Parenthesis

Sometimes you need to group sub-expressions with parenthesis, but do not want the parenthesis to spit out another marked sub-expression, in this item a non-marking parenthesis (?:expression) can be used. For example the following expression creates no sub-expressions: ( ?:abc) *

Forward Lookahead Asserts

There are two forms of these; one for positive forward look-ahead asserts, and one for negative look-ahead asserts:

Regex: Description
(?=abc): Positive Lookahead. Matches zero characters only if they are followed by the expression abc.
(?!abc): Negative Lookahead. Matches zero characters only if they are not followed by the expression abc.

Lookbehind Asserts

There are two forms of lookbehinds: postitive lookbehind and negative lookbehind.

Regex: Description
(?<=pattern): Positive lookbehind. Matches zero characters only if they are preceded by the expression defined by the pattern.
(?<!pattern): Negative lookbehind. Matches zero characters only if the are not preceded by the expression defined by the pattern.

When defining the pattern for the lookbehind must be a fixed length. The . and ? wildcards can be used, but * and + are not supported.

Combining Multiple Lookbehinds

Multiple lookbehinds can be combined using the | operator. You cannot combine several expression using or in the statement within the (?<=pattern) expression. Instead, you must use a non-marking parenthesis:

(?:(?<=abc)|(?<=def)|(?<!xyz))

Alternatives

Alternatives occur when the expression can match either one sub-expression or another, each alternative is separated by a "|", or a "\|" if the flag regbase::bk_vbar is set, or by a newline character if the flag regbase::newline_alt is set. Each alternative is the largest possible previous sub-expression; this is the opposite behavior from repetition operators.

Examples

Regex: Description
a(b|c): Matches 'ab' or 'ac'.
abc|def: Matches 'abc' or 'def'.

Sets

A set is a set of characters that can match any single character that is a member of the set. Sets are delimited by "[" and "]" and can contain literals, character ranges, character classes, collating elements and equivalence classes. Set declarations that start with "^" contain the compliment of the elements that follow.

Examples

Character literals:

Regex: Description
[abc]: Will match either of 'a', 'b', or 'c'.
[^abc]: Will match any character other than 'a', 'b', or 'c'.

Character ranges:

Regex: Description
[a-z]: Will match any character in the range a to z.
[^A-Z]: Will match any character other than those in the range A to Z.

Character classes are denoted using the syntax [:classname:] within a set declaration, for example [:space:]] is the set of all white-space characters. Character classes are only available if the flag regbase::char_classes is set. The available character classes are:

Classname: Description
alnum: Any alpha numeric character.
alpha: Any alphabetical character a-z and A-Z. Other characters may also be included depending upon the locale.
blank: Any blank character, either a space or a tab.
cntrl: Any control character.
digit: Any digit 0-9.
graph: Any graphical character.
lower: Any lower item character a-z. Other characters may also be included depending upon the locale.
print: Any printable character.
punct: Any punctuation character.
space: Any white space character.
upper: Any upper item character A-Z. Other characters may also be included depending upon the locale.
xdigit: Any hexadecimal digit character, 0-9, a-f and A-F.
word: Any word character - all alphanumeric characters plus the underscore.

Regex: Description
\w: Use in place of [:word:]
\s: Use in place of [:space:]
\d: Use in place of [:digit:]
\l: Use in place of [:lower:]
\u: Use in place of [:upper:]

Collating elements take the general form [.tagname.] inside a set declaration, where tagname is either a single character, or a name of a collating element, for example [.a.]] is equivalent to [a], and [.comma.]] is equivalent to [,]. The library supports all the standard POSIX collating element names, and in addition the following digraphs: "ae", "ch", "ll", "ss", "nj", "dz", "lj", each in lower, upper and title item variations. Multi-character collating elements can result in the set matching more than one character, for example [.ae.]] would match two characters, but note that [^[.ae.]] would only match one character.

Equivalence classes take the general form [=tagname=] inside a set declaration, where tagname is either a single character, or a name of a collating element, and matches any character that is a member of the same primary equivalence class as the collating element [.tagname.]. An equivalence class is a set of characters that collate the same, a primary equivalence class is a set of characters whose primary sort key are all the same (for example strings are typically collated by character, then by accent, and then by item; the primary sort key then relates to the character, the secondary to the accentuation, and the tertiary to the item). If there is no equivalence class corresponding to tagname, then [=tagname=] is exactly the same as [.tagname.]. Unfortunately there is no locale independent method of obtaining the primary sort key for a character, except under Win32. For other operating systems the library will "guess" the primary sort key from the full sort key (obtained from strxfrm ), so equivalence classes are probably best considered broken under any operating system other than Win32.

To include a literal "-" in a set declaration then: make it the first character after the opening "[" or "[^", the endpoint of a range, a collating element, or if the flag regbase::escape_in_lists is set then precede with an escape character as in "[\-]". To include a literal "[" or "]" or "^" in a set then make them the endpoint of a range, a collating element, or precede with an escape character if the flag regbase::escape_in_lists is set.

Line anchors

An anchor is something that matches the NULL string at the start or end of a line: "^" matches the NULL string at the start of a line, "$" matches the NULL string at the end of a line.

Back references

A back reference is a reference to a previous sub-expression that has already been matched, the reference is to what the sub-expression matched, not to the expression itself. A back reference consists of the escape character "\" followed by a digit "1" to "9", "\1" refers to the first sub-expression, "\2" to the second etc. For example the expression (.*)\1 matches any string that is repeated about its mid-point for example abcabc or xyzxyz. A back reference to a sub-expression that did not participate in any match, matches the NULL string: NB this is different to some other regular expression matchers. Back references are only available if the expression is compiled with the flag regbase::bk_refs set.

Characters by code

This is an extension to the algorithm that is not available in other libraries, it consists of the escape character followed by the digit "0" followed by the octal character code. For example "\023" represents the character whose octal code is 23. Where ambiguity could occur use parentheses to break the expression up: "\0103" represents the character whose code is 103, "(\010)3 represents the character 10 followed by "3". To match characters by their hexadecimal code, use \x followed by a string of hexadecimal digits, optionally enclosed inside {}, for example \xf0 or \x{aff}, notice the latter example is a Unicode character.

Word operators

The following operators are provided for compatibility with the GNU regular expression library.

Regex: Description
\w: matches any single character that is a member of the "word" character class, this is identical to the expression [:word:]].
\W: matches any single character that is not a member of the "word" character class, this is identical to the expression [^[:word:]].
\<: matches the NULL string at the start of a word.
\>: matches the NULL string at the end of the word.
\b: matches the NULL string at either the start or the end of a word.
\B: matches a NULL string within a word.

The start of the sequence passed to the matching algorithms is considered to be a potential start of a word unless the flag match_not_bow is set. The end of the sequence passed to the matching algorithms is considered to be a potential end of a word unless the flag match_not_eow is set.

Buffer operators

The following operators are provide for compatibility with the GNU regular expression library, and Perl regular expressions:

Regex: Description
\`: matches the start of a buffer.
\A: matches the start of the buffer.
\': matches the end of a buffer.
\z: matches the end of a buffer.
\Z: matches the end of a buffer, or possibly one or more new line characters followed by the end of the buffer.

A buffer is considered to consist of the whole sequence passed to the matching algorithms, unless the flags match_not_bob or match_not_eob are set.

Escape operator

The escape character "\" has several meanings.

Inside a set declaration the escape character is a normal character unless the flag regbase::escape_in_lists is set in which item whatever follows the escape is a literal character regardless of its normal meaning.
The escape operator may introduce an operator for example: back references, or a word operator.
The escape operator may make the following character normal, for example "\*" represents a literal "*" rather than the repeat operator.

Single character escape sequences

The following escape sequences are aliases for single characters:

Regex: Description
\a: Character code: 0x07 Bell character
\f: Character code: 0x0C Form feed
\n: Character code: 0x0A Newline character
\r: Character code: 0x0D Carriage return
\t: Character code: 0x09 Tab character
\v: Character code: 0x0B Vertical tab
\e: Character code: 0x1B ASCII Escape character
\0dd: Character code: 0dd An octal character code, where dd is one or more octal digits.
\xXX: Character code: 0xXX A hexadecimal character code, where XX is one or more hexadecimal digits.
\x{XX}: Character code: 0xXX A hexadecimal character code, where XX is one or more hexadecimal digits, optionally a unicode character.
\cZ: Character code: z-@ An ASCII escape sequence control-Z, where Z is any ASCII character greater than or equal to the character code for '@'.

Miscellaneous escape sequences:

The following are provided mostly for Perl compatibility, but note that there are some differences in the meanings of \l \L \u and \U :

Regex: Description
\w: Equivalent to [:word:]]
\W: Equivalent to [^[:word:]]
\s: Equivalent to [:space:]]
\S: Equivalent to [^[:space:]]
\d: Equivalent to [:digit:]]
\D: Equivalent to [^[:digit:]]
\l: Equivalent to [:lower:]]
\L: Equivalent to [^[:lower:]]
\u: Equivalent to [:upper:]]
\U: Equivalent to [^[:upper:]]
\C: Any single character, equivalent to ' . '
\X: Match any Unicode combining character sequence, for example a\x 0301 (a letter a with an acute)
\Q: The begin quote operator, everything that follows is treated as a literal character until a \E end quote operator is found
\E: The end quote operator, terminates a sequence begun with \Q

What gets matched?

The regular expression library will match the first possible matching string, if more than one string starting at a given location can match then it matches the longest possible string, unless the flag match_any is set, in which item the first match encountered is returned. Use of the match_any option can reduce the time taken to find the match - but is only useful if the user is less concerned about what matched - for example it would not be suitable for search and replace operations. In list where their are multiple possible matches all starting at the same location, and all of the same length, then the match chosen is the one with the longest first sub-expression, if that is the same for two or more matches, then the second sub-expression will be examined and so on.